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ABSTRACT 

This paper summarizes and highlights some of the 
critical developments, issues, and trends which impact the work of 
survey research. The issue of preventing or minimizing nonresponse 
bias by successfully obtaining cooperation from people being surveyed 
continues to be of concern to survey research professionals. The most 
effective response rate stimulating strategies are those that are 
creatively designed to fit the people being surveyed and flexible 
enough to meet the changing characteristics of different populations. 
Regarding the effect of people who are unreachable by telephone, the 
important issue is making certain the sample is not different from 
the population in some way that will affect the results. An approach 
to obtaining truthful answers to questions of a sensitive nature is 
the randomized response technique, which presents the respondent with ^ 
more questions then just the sensitive question. New developments in 
technology have impacted survey research in data collection and data 
analysis, specifically with the use of Computer Assisted Telephone 
Interviewing and the personal computer. Certain emerging trends might 
endanger the quality of research efforts: (1) over-emphasizing quick 
results: (2) allowing technology to substitute for human quality; (3) 
restricting options; and (4) misusing qualitative methods. (PN) 
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INTRODUCTION 



The purpose of this presentation is to summarize and highlight some of the 
critical developments, issues, and trends which are impacting the work of survey 
research professionals. Advances in technology, continually advancing methodological 
knowledge, and changes in the social environment in which survey researchers work 
have all contributed to the changes which have occurred. 

The presentation will begin with a brief discussion of information collected for 
the second edition of "Questionnaires: Design and Use." Also included in the 
presentation will be a discussion of several new developments which are having a 
major impact on survey research, and discussion of issues of which survey research 
professionals must be aware as they conduct their work in an environment that is 
continuing to change at an increasingly rapid pace. 

Response Rate Issues 

The issue of preventing or minimizing nonresponse bias by successfully obtaining 
cooperation from people being surveyed continues to be of concern to survey research 
professionals (9, 28). Without going into a lengthy review of the background of the 
many strategies that have been recommended over the years as response rate 
stimulators, there are a few more rbcent developments which deserve attention. 
These include claims that the number of people who refuse to participate in surveys is 
increasing, issues related to the definition of response rate, the need for creativity and 
flexibility in designing strategies to motivate potential respondents, and research 
being done to quantify and predict patterns of responding to questionnaires. 

There appears to be a growing tendency among many survey research profes- 
sionals to endorse a belief that people are becoming increasingly resistant to surveys. 
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The argument is usually based on assumptions that surveys are becoming more 
common, people are being surveyed more frequently, and, therefore, the number of 
••hard-core" nonrespondents is rapidly increasing. Fortunately, this belief is being 
challenged (23). 

In more than 15 years of conducting surveys by mail, telephone, and in person 
v^ith all types of persons ranging from the least to the most educated, from 
unemployed to professionals, including persons of varied races, national origin, and 
geographic locations, the author has routinely achieved response rates between 90% 
and 100% of the originally selected sample without replacing anyone in the sample. 
This experience has revealed no evidence that respondent resistance is increasing. 
Perhaps people are choosing to live their lives in ways that cause them to be less 
available to survey researchers, and perhaps people are becoming intolerant of poor 
quality questionnaires administered by incompetent interviewers or, worse yet, by 
automated machines. However, if a well designed survey is properly presented to 
potential respondents, they will cooperate. 

Survey researchers must commit themselves to the belief that people will 
respond and then direct their energies toward structuring a strategy to obtain 
cooperation. Effort will be far more productively spent this way than in adopting a 
defeatist attitude and then attempting to devise ways to compensate or adjust for low 
response rates. 

Another issue related to response rate that needs attention is the way in which 
response rate is conceptualized by some survey researchers. One of the more 
disturbing practices is the difference between the way people frequently present 
response rates from mail and telephone surveys. When discussing mail surveys, 
response rate is usually defined as the percentage of mailed questionnaires that are 
completed and returned. However, in studies using telephone interviews, it is not 
uncommon for researchers to replace people who are not reachable with people who 
are reachable until the desired number of interviews are completed. This practice 



ignores the problem of nonresponse bias as illustrated in the comparison below: 



Situation 1 

Original sample size 
Responses received 
Response rate = 200 
300 



Situation 2 

300 Original sample size := 300 

200 Responses received = 200 

67% Responses received after 

replacements =300 
Claimed response rate = 100% 
Actual response rate = unknown 

(without knowing how 
many replacements 
were made) 



What is needed is a uniform manner for computing and reporting response rates 
for all surveys whether mail, telephone, or personal interviews. One reasonable 
approach is to define response rate as the percentage of completed surveys obtained 
from the original sample without replacement (3). The only replacements permitted 
would be those who do not fit the population definition. An example would be 
replacing people who have moved out of the United States if the population being 
studied is defined as people living in the United States. Of course, another perplexing 
issue is deciding what constitutes a "completed" survey — must every item be 
answered? how about three-fourths of the items? or one-half? One solution to this 
dilemma is to calculate and report a response rate for each item contained on the 
survey questionnaire. This will minimize data lost due to exclusion of less than totally 
complete surveys and will allow the users of research results to decide how much 
credibility to assign to the data derived from each item. For example, in one survey, 
most questions may have response rates of 90% or more. However, one or two items 
may be poorly written and have low response rates; perhaps the results from these 
items should be ignored or at least given less credibility than the rest of the results. 

Research is continuing to examine various ways to stimulate response rates (8, 
11, 12, 14, 15) and researchers are continuing to study the effects of various types of 
mail, cover letters, postage, incentives, and many other variables which have been 
examined for some time. However, there are also new directions being investigated. 



For example, a number of studies have examined the effect of obtaining some 
type of prior commitment from a potential respondent before the actual survey (14, 
18, 19, 24). This initial commitment could take the form of a brief postcard to be 
answered and returned, a brief telephone solicitation, or other forms. This technique 
has come to be appropriately known as the "foot-in-the-door" technique. 

A number of studies are working toward quantifying various aspects of the 
response rate issue. One 1980 study (20) attempts to quantify the amount of time 
required to obtain various levels of response rate in mail surveys by developing a 
mathematical function and using regression to predict response rates. 

When examining the various studies designed to test the effect of various 
specific treatments on response rate, one cannot help notice that results are often 
conflicting, i.e., what works for one study does not work for another study. Actually, 
this should be neither surprising nor alarming. After all, survey research is totally 
dependent on people to provide results, and we all know people are not all the same 
and do not all react the same way to similar circumstances. We should be grateful for 
this — if people were all exactly the same, survey research would be unnecessary and 
we would all be unemployed. 

As researchers work to define variables that affect response rates more 
precisely and attempt to quantify these effects, there may be a tendency to accept 
the findings of a study as "fact" and assume the results will be the same for all survey 
research studies. A better and more productive approach is to use the various 
techniques to build a repertoire of techniques and be prepared to use them when 
appropriate. This will allow tailoring the approach to be most effective with the 
population being studied. The most effective response rate stimulating strategies are 
those that are creatively designed to fit the people being surveyed and flexible to 
meet the changing characteristics of different populations (1). 



Effect of People Who Are Unreachable by Telephone 

One issue that continues to be of concern to survey researchers who use 
telephone interviewing as a data collection technique, is the question of the extent of 
the bias caused by people who are unreachable by telephone. In thinking about this 
issue, it is important to remember that there are three major reasons why people are 
not reachable by telephone: (1) they do not have telephone service in their home or 
accessible to them, (2) they have telephone service, but have an unpublished number, 
or (3) they have phone service and a published number, but are difficult to contact. 

The third group, i.e., those with phone service who are difficult to contact 
present a problem that is different from the other two groups. This group really 
presents a challenge of nonresponse and the threat they present to quality data is 
nonresponse bias. Therefore, the solution for handling this group is no different from 
any other group that is difficult to get to respond whether in a mail survey, a 
telephone interview, or a personal interview. The key is to make contact by persistent 
and carefully scheduled attempts, and by trying different contact methods, e.g. mail, 
personal visits, and telephone. For example, telephone calls can be scheduled on 
different days of the week and at different times of the day, a postcard or letter 
indicating your need to contact them could be sent including a name and phone number 
they can call to schedule a convenient interview time, or a visit could be made to talk 
with neighbors to determine when the potential respondent is reachable. 

The other two groups present a different problem. In discussing the possible bias 
introduced by these two groups, people often cite research which shows that 
households without phone service differ in certain ways from households with phone 
service and that households with unpublished phone numbers differ from those with 
published phone numbers (5, 16, 17). Such studies frequently rely on comparisons of 
demographic characteristics. 

It has become trendy to argue that all telephone surveys should use a random 
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digit dialing procedgre to prevent the bias that can result from missing unpublished 
telephone numbers (4). Before deciding such a procedure is necessary and incurring 
additional expense implementing it (22), consider the effect these problem groups v/ill 
have on the outcome of the survey. 

The important concern for the survey researcher is how much do these 
differences affect the results of the survey being conducted? Surveys are usually 
conducted by selecting a sample and then projecting the reults obtained from that 
sample to the population from which the sample was selected. The important issue is 
making certain the sample is not different from the population in some way that will 
affect the results. 

Therefore, the focus of concern for survey researchers should be not whether 
households with telephones differ from those without telephones or whether those with 
or without published listings diff er? but, rather, whether the total sample results differ 
when such groups are included or not included in the sample. In most studies, the 
percentage of sample that are in these two groups will be small and have little effect 
on the survey results. Studies designed to examine the size of such effects have found 
that, while differences between these groups and other respondents may be statisti- 
cally significant the effect they have on changing the overall survey results is small — 
often 2% or less (21, 23, 26, 29). 

Obviously, these results should not be interpreted to mean that one need never 
worry about bias caused by households without telephones or with unpublished 
telephone numbers. In some studies an error of 1% or 2% or less may not be tolerable, 
or in studies of certain segments of a population such as very low income groups many 
people may not have telephones and the effect may be large. Decisions as to the 
necessity of using random digit dialing procedures must be made for each study. 
However, it does appear that survey researchers need not uniformly adopt random 
digit dialing procedures for all telephone studies. 
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Rancbmized ResporiBe Technique 

Obtaining truthful answers to questions of a sensitive nature is a problem which 
faces nearly every survey research professional at some time. One approach to this 
problem that is receiving positive results is the randomized response technique. 
Basically, this technique involves presenting the respondent with more questions than 
just the sensitive question. The respondent answers only one question and is given 
some random means such as a coin to flip or die to toss to select which question will 
be answered. Only the respondent will know which question was answered making it 
impossible to determine if the sensitive question was answered. The respondent's 
answer is then combined with answers from other respondents and, knowing the 
probability of answering the sensitive question, summarized responses to the sensitive 
question can be computed. 

Most researchers studying this technique have found it lo be successful in 
obtaining answers to sensitive questions (6, 13, 30). 

Technolo^cal Developments 

New developments in technology are definitely having an impact on survey 
research. The two activities in survey research that have been most directly affected 
so far are data collection and data analysis. 

The most obvious change in data collection technology is tlie exploding increase 
in the use of Computer Assisted Telephone Interviewing (CATI). CATI systems first 
emerged in the mid-1970's and have continued to increase in popularity and sophistica- 
tion. These systems utilize a computer to store and display an interview questionnaire 
so the interviewer reads the questions displayed on the monitor screen. In the early 
versions, the interviewer used a keyboard to enter responses; however, systems now 
have been developed to eliminate the keyboard by using light (.ens touched to the 
screen. 



CATI systems have many advantages including the convenience of having the 
computer pre-programmed to control the flow of questions to prevent missing of skip 
patterns. They can also "edit" the interviewer's work to prevent mistakes such as 
entering an inappropriate response. These features make it possible to administer very 
complex questionnaires with skip patterns that would be cumbersome and prone to 
more error using paper and pencil forms. Also, CATI systems make it easier for 
interviewers to incorporate responses from questions into later questions, such as 
inserting names of people, dates, brand names, etc. CATI systems also permit visual 
monitoring of interviewers' work in addition to the usual auditory monitoring. 
Responses are instantaneously entered into the computer memory greatly speeding up 
the research process by eliminating later data entry and much of the usual editing 
process. CATI systems are now available for use with multi-user computers utilizing a 
host computer and remote terminals or for stand-alone personal computers. 

In addition to controlling display of questionnaire items and entry of responses, 
software for CATI systems can no*.v also assist with survey management by keeping 
track of interview attempts and controlling the scheduling of callback attempts at 
pre-s^pecified times. Careful scheduling of callbacks can help reduce nonresponse. 
Recording of response rates, refusals, and other information can also be easily 
accomplished. 

Of course, there are limitations. At present, most CATI systems are quite 
expensive to install unless the hardware is already available. Also, many systems are 
not as flexible as one might wish them to be. For example, the recording of open- 
ended responses cannot always be accommodated. Some CATI systems cannot display 
questions as quickly as desirable and such delays can be bothersome to the smooth flow 
of an interview administration. In addition, some interviewers may have trouble 
adjusting to CATI because they can write faster than they can type or they make more 
errors in recording responses because of limited keyboard skills. 

Although CATI systems are becoming quite common and their use is expanding, 
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there has not been sufficient research conducted to measure the positive and negative 
effects of CATI on interview results. More carefully designed studies are needed to 
compare the results obtained with a CATI system to results obtained with paper and 
pencil interviewing. One such study (28) found that CATI resulted in a large reduction 
in the number of interviewing errors which necessitate recontacting respondents. 

The other significant technological catalyst in survey research is the personal 
computer. It has affected many phases of the research process, but its largest impact 
has been in the analysis of data. Personal computers have also assisted with other 
aspects of the study. They have become a marvelous tool for managing a study. 
Inexpensive data base management software is being used to assist with sample 
selection, to structure sampling frames, and to systematically keep track of responses 
Ko mail surveys. Elaborate tracking systems can be developed quite inexpensively to 
record returned questionnaires and to indicate daily which people haven't responded 
and who should receive follow-up contact. 

The availability of easy*to-use, inexpensive software for statistical analysis is 
mushrooming. Perhaps the most significant developments for survey researchers is the 
availability of familiar software packages such as SPSS and SAS for the personal 
computer. SPSS can be purchased for approximately $800 and additional multivariate 
procedures and graphics packages can be added at little additional cost. This means 
that a complete system of hardware, such as an IBM PC XT, and SPSS software can be 
; purchased for as little as several thousand dollars. The speed and capacity of these 

\ programs is very satisfactory for almost any survey research project. 

Trends 

« 

The changing environment in which survey research is developing is presenting 
researchers with great opportunity. However, along with the opportunities tempting 
dangers are also present. Researchers and managers of research must avoid being 
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easily seduced into embracing certain emerging trends which will endanger the quality 
of research efforts. These twentieth century sirens of the research world include 
over-emphasis on quick results, allowing technology to substitute for human quality, 
tendencies to permit expensive investments in technology and veneration of experts to 
restrict our choice of options, and the misuse of qualitative research methods often in 
the interest of reducing expense. 

A. Over-Emphasis on Quick Results 

In an age in which computers can create art and produce music, in which we see 
major news events unfold live as they happen on the evening news, and in which the 
results of presidential elections are known before the last voters have cast their vote, 
it should not be too surprising that users of research results are seeking results faster 
and faster. 

Some reduction in the time required to conduct a survey research project is not 
an unrealistic expectation. Perhaps the biggest time saver is technology. Faster and 
more accessible computers and statistical analysis software, word processors to reduce 
the time required to prepare questionnaires and written reports, and CATI have all 
helped speed up the research process. Also, non-technological improvements in survey 
research such as improved knowledge concerning how to motivate people to respond to 
surveys, and judicious use of correct sampling strategies which can reduce the number 
of people that must be surveyed can result in time savings. 

However, we must recognize that the purpose of conducting survey research is to 
obtain accurate and dependable results. To accomplish this goal and not compromise 
results requires recognition of the fact that the time required to conduct good survey 
research can only be compressed to certain limits given our current knowledge and 
technology. 

For example, to the uninitiated, a plausible solution to the "time crunch" might 
be to expand the size of the interviewing staff so more interviews can be conducted 




in less time. This logic, carried to extreme, could be interpreted to suggest that, 
given enough interviewers, even a large sample of many thousands could be surveyed in 
less than one day. However, to have accurate results, sufficient time must be taken to 
permit call backs of people not initially available. This usually requires scheduling 
attempts at different times of the day and on different days of the week, thereby 
necessitating aL least several days or even weeks to contact the original sample and 
avoid massive replacements in the sample which can seriously bias the results. 

Similarly, people conducting mail surveys may suggest mailing of massive 
numbers of questionnaires to obtain the required number of responses in a shorter 
time. This argument overlooks the need for a high response rate and can, again, 
sacrifice quality unnecessarily. 

While time can be condensed, survey researchers must not allow themselves to 
be fooled or required to conduct research in time constraints that are not realistic at 
present and that will impede the quality of the research. As professionals, we must 
educate the consumers of our product, i.e., research results, about the time required 
to conduct good research and about the price that must be paid for excessive time 
compression. 

B. A Caution About Technology 

The technological revolution has certainly had a major impacl on the field of 
survey research. These impacts are occurring in all phases of survey work. The most 
obvious example is in analysis of responses — personal computers are now within 
nearly everyone^s reach together with inexpensive and comprehensive software. 
Computer technology in the form of word processors has impacted questionnaire 
design and report preparation. Sampling is frequently done by using computer 
generated random digits and using data base management software. Data is often 
collected using computer assisted interviewers. Other technological changes which 
will impact survey research include electronic mail, telephone answering machines. 



home computers integrated with communications systems, etc., etc. 

There is a danger that fascination with these changes may cause researchers to 
forget that these marvels are merely tools to help them do their work. At this point in 
time, the human element is critically important and essential to the survey research 
process. For example, computer assisted interviewing can greatly reduce Interviewer 
errors in recording responses; however, some people seem to think they can utilize 
lesser quality interviewers and that the computer will compensate. Nothing can 
replace the skill and resourcefulness of a good, experienced, trained, professional 
interviewer. Similarly, nothing can replace the analytical mind of a skilled data 
analyst. Mountains of useless computer print-outs are being generated by people using 
personal computers to compute statistical procedures that are unnecessary because 
the user doesn't know what procedures to do. 

The advances in technology must be used to enhance and expand the capabilities 
and capacity of skilled research personnel, not substitute for them. 
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C. Restricting Options 

Another disturbing trend in survey research is what appears to be a diminishing 
creativity and flexibility among some researchers. This is caused by at least two 
factors. First, researchers have often invested heavily in expensive new machinery 
and software. Once such an investment is made, it must be used. Therefore, some 
researchers with expensive CATI systems feel obligated to use telephone interviewing 
to conduct every survey research project they do. People may also invest heavily in 
the design of a questionnaire and then try to recover that investment by repeated use 
of the same questionnaire. Unfortunately, many people use these "generic" question- 
naires in situations for which they were not designed with the result that data is often 
overly general and sometimes not the data needed to accomplish the intended research 
purpose. 

Second, researchers must be careful not to let their creativity be stifled by 
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••veneration of experts." It is tempting to look for Uie method to use in all situations 
and then mass produce it* Indeed attempts have been made to develop such a method 
or to create one out of the recommendations of researchers (10). 

Collections of questionnaire items have been developed and, while this practice 
can be an assistance to questionnaire construction, survey research professionals must 
be cautious in their use. Overreliance on such resources can lead to acceptance of 
items as "high qualit/' items because they were developed by "experts." Only careful 
examination of the methods used to develop these items such as pretest procedures, 
information about the populations with which they have been used, etc. will provide 
needed information about their quality. If such items are used in questionnaire 
development, it is essential to pretest them with the population who will respond to 
the questionnaire to be certain they will be successful with that specific population. 

Also, "borrowed" items can be dangerous if they are used as a "short cut" to 
questionnaire construction resulting in insufficient time and energy being devoted to 
clarifying the specific goals for the survey. This can result in data that is less useful 
than desired because the items may not address the exact purposes of the survey 
research. 

However, the essence of survey research is collecting information from people. 
People are not all the same; they are diverse in background, interests, and motiva- 
tions. Research methods must recognize this and researchers would be well advised to 
amass an arsenal of techniques at their disposal and broaden their options through 
creative thinking and methodological research, rather than narrowing their options by 
seeking one easy solution that can be applied to all situations. 

D. Misuse of Qualitative Methods 

The use of qualitative research methods, especially focus groups, is expanding. 
Many people see these as an inexpensive alternative to quantitative studies. People 
sometimes fail to recognize that qualitative and quantitative methods are not the 
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same and are not intended to achieve the same results. Qualitative methods are not 
cheap substitutes for quantitative studies^ but| rather^ are useful techniques when used 
correctly. 

One author (7) has summarized the appropriate uses of focus groups into three 
categories: (1) exploratory approach, (2) clinical approach, and (3) phenomenologlcal 
approach. The exploratory approach uses focus groups to generate ideas to help 
develop further research or to test ideas and procedures. The clinical approach uses 
focus groups as a means for an "expert" to observe people's behavior and then form 
clinical judgements which form the results. The phenomenologlcal approach empha- 
sizes the value to be gained from observing the interaction of focus group participants 
and bases results on information obtained from studying those interactions. 

Survey researchers must be aware when each of these methods is appropriate and 
assist the research consumer in learning the distinction also. Generally, focus groups 
do not provide data that are generalizable to populations with any measurable degree 
of accuracy. If such generalization is desired, a quantitative study is necessary. 
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